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Feasibility  Report  for 

Experimental  Inter-service  Computer- Administered  Testing  for  the 
Armed  Services  Vocational  Aptitude  Battery  (ASVAB) 

Adjuvant  Consulting  Inc.  (ACI)  provided  computer  hardware  and  technical  personnel  to 
document  Lessons  Learned  for  facilities  support  services  to  conduct  short-term  experimental 
(non-recurring)  trials  of  computerized  testing.  This  endeavor  included  three  experimental  testing 
sessions.  Each  testing  session  was  comprised  of  100  recruits  in  two  3-hour  sessions.  Rented 
laptops  were  used  to  administer  these  sessions.  ACI  performed  the  following  activities  during 
this  assessment  period: 

•  Set  up  computers  from  bulk  storage  to  test  subject  desktops  including  appropriate 
electrical  connections  and  computer  preparation  for  experimental  testing  from 
USB/flash  drive. 

•  Observed  and  documented  logistical  problems  and  made  recommendations  for 
improving  the  technical  delivery  aspects  of  the  experimental  testing  protocols. 

•  Provided  computer  corrective  actions  for  technical  problems,  documented 
recovery  methods,  or  alert  the  test  content  proctor  for  software  malfunction 
issues. 

•  Secured  computers  before  sessions,  during  lunch  breaks,  and  after  final  testing 
sessions. 

•  Used  the  test  content  proctor"s  instructions  for  capturing  test  subject  data  from 
each  experimental  testing  station  onto  a  standalone  proctor  PC  and  documented 
procedures. 

•  After  the  third  exemplar  testing  session,  returned  all  experimental  testing 
materials  back  to  their  appropriate  custodian. 

•  Provided  the  final  report  (Lessons  Learned)  for  review  to  AFPC/DPSF. 


The  following  sections,  Equipment  Storage,  Power  Demands,  Testing  Station  Build, 
Testing  Station  Teardown,  Test  Administration,  and  Data  Collection  categorize  the  Fessons 
Learned  gathered  throughout  this  assessment  period. 
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Equipment  Storage: 


Equipment  storage  could  represent  a  huge  logistical  problem.  Every  effort  should  be 
taken  to  keep  the  equipment  (Laptops,  surge  protectors,  and  extension  cords)  on  location  at  the 
assessment  site.  Equipment  now  becomes  assessable  to  multiple  individuals  and  reduces  the 
possibility  of  a  single  point  of  failure. 

The  transport  of  the  equipment  from  outside  storage  to  the  testing  facility  would  be 
extremely  labor  intensive  and  would  require  additional  implementation  costs.  In  addition,  storage 
of  equipment  at  a  non  government  facility  could  present  unnecessary  security  risks  that  could 
have  serious  impacts  on  the  delivery  of  the  assessment.  Every  opportunity  should  be  taken  to 
minimize  these  risks  to  provide  dependable  and  consistent  service  for  each  of  the  assessment 
sessions. 

Onsite  storage  of  the  equipment  was  beneficial  in  this  instance  because  it  allowed  Air 
Force  Personnel  at  the  BMT  Reception  Center  access  to  move  the  equipment  from  the  2nd  floor 
secured  storage  to  the  1st  floor  testing  area  before  the  Testing  Station  Build  step  was  started. 


Power  Demands: 


The  test  site  should  be  inspected  in  conjunction  with  the  Test  Station  Configuration 
Diagram.  (Attachment  I)  This  diagram  contains  the  planned  seating  arrangements  for  each  table 
within  the  test  and  can  provide  a  good  source  of  reference  to  determine  the  test  facilities  power 
requirements.  A  Night-Light  can  be  used  to  test  each  wall  outlet  (Left  and  Right)  to  ensure  each 
side  is  operational.  This  should  be  done  for  all  the  outlets  within  the  testing  configuration.  If 
possible,  determine  the  circuit  breaker  in  the  Electrical  Box  that  you  are  using  for  the  testing 
tables.  This  will  be  valuable  infonnation  to  have  in  event  of  a  power  outage.  As  backup,  plan 
which  outlets  will  be  used  to  reroute  power  if  an  overload  situation  is  encountered.  These  should 
be  outlets  on  different  circuit  breakers  than  the  testing  tables.  Determine  the  distance  from  the 
test  tables  to  the  alternate  power  source  to  ensure  that  power  can  be  rerouted.  Alternate  power 
can  also  be  acquired  through  an  external  power  source  such  as  a  power  generator.  These 
additional  steps  if  taken  can  minimize  your  downtime  and  provide  the  consistency  needed  to 
conduct  the  pilot  testing  and  produce  valid  test  metrics. 

If  the  test  facility  is  a  building  that  provides  services  to  others  such  as  a  reception  or 
convention  center  a  schedule  or  calendar  of  daily  events  should  be  acquired.  This  could  have 
serious  implication  on  the  power  demands  of  the  building.  What  works  one  week  can  fail  the 
next  so  plan  ahead  to  ensure  that  power  is  available  and  uninterrupted  for  the  duration  of  each 
testing  session.  A  stable  test  environment  is  necessary  to  provide  the  measurements  need  to 
build  an  accurate  test  instrument. 


2 


Testing  Station  Build: 


Test  tables  should  be  placed  in  conjunction  with  the  wall  power  outlets  to  minimize  the 
need  for  extension  cords.  Surge  protectors  should  be  at  least  eight  (8)  feet  in  length  with  no  less 
than  six  (6)  outlets  to  allow  for  adequate  space  from  the  table  to  the  power  source.  This  will 
render  a  cleaner  table  setup  and  provide  good  seating  space  for  the  test  candidates.  Each  table  is 
configured  of  four  (4)  Laptops.  One  wall  outlet  should  provide  adequate  power  for  eight  (8) 
laptops.  This  will  allow  for  single  tables  to  be  linked  together  via  surge  protector,  to  extension 
cord,  then  to  surge  protector.  Various  combinations  can  be  used  based  on  the  table  configurations 
of  the  testing  facility.  However,  no  more  than  eight  (8)  Laptops  per  right  or  left  wall  receptacle 
are  recommended.  This  is  running  heavy  but  should  provide  adequate  power  providing  no  other 
power  drain  is  expected. 

Setup  of  twenty  five  (25)  tables  and  transport  of  nineteen  (19)  boxes  containing  Laptops, 
Surge  Protectors,  and  Extension  Cords  from  and  to  storage  was  provided  by  Air  force  personnel 
on  detail  at  the  BMT  Reception  Center. 


Testing  Station  Teardown: 


When  working  with  such  a  large  number  of  testing  stations  it  is  helpful  to  make  the 
teardown  of  the  equipment  as  streamlined  as  possible.  Therefore,  it  is  recommended  that  the 
Laptop  Case  be  left  open  (unzipped)  after  the  test  station  setup  is  completed.  The  empty  case 
should  be  kept  near  the  table  with  the  station.  There  should  be  four  (4)  empty  and  unzipped 
cases  per  table.  After  the  candidate  completes  their  test,  the  proctor  should  follow  the  shut  down 
procedures.  If  this  is  the  final  test  of  the  day  then  shutdown  windows,  close  Laptop  cover,  and 
remove  thumb  drive.  Place  an  empty  case  under  each  laptop  to  prepare  for  table  teardown. 


Test  Administration: 


ACT's  role  in  the  administration  of  the  test  was  limited  to  the  functions  of 
Observer/Computer  Information  Analyst  and  Senior  Analyst.  This  was  achieved  with  great 
success.  In  addition  to  these  tasks  ACI  was  asked  to  assist  as  proctors.  As  proctors  ACI  answered 
questions  the  candidate'^  presented  to  them  and  assisted  in  managing  the  flow  of  candidates  to 
and  from  the  testing  area.  These  activities  were  documented  on  the  “Test  Session  Record”  and 
returned  to  test  administrators  at  the  completion  of  each  testing  session. 

It  is  recommended  that  each  individual  functioning  as  a  proctor  be  given  the  opportunity 
to  take  the  test  themselves.  This  would  establish  familiarity  with  the  test  and  assisted  in 
providing  accurate  responses  to  the  candidate"s  questions.  These  are  a  few  samples  of  the 
questions  asked  during  the  administration  of  the  test: 

1 .  It  seems  like  I"ve  answered  these  questions  before.  Am  I  taking  the  test  again? 
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2.  Do  we  get  scratch  paper? 

3.  Do  I  have  to  answer  all  questions? 

4.  Can  I  skip  questions  if  I  don"t  know  the  answer? 

5.  Why  am  I  just  on  the  first  session?  I'Ve  be  testing  for  a  while.  Seems  like  I  should  be 

further  along. 

These  types  of  questions  can  be  addressed  as  part  of  the  greetings  script  given  to  the 
candidates  at  the  start  of  each  testing  session. 

Candidates  had  difficulty  remaining  awake  during  the  morning  and  afternoon  sessions.  It 
is  recommended  that  test  candidates  be  allowed  to  remove  their  jackets  and  stand  if  fatigued. 
These  instructions  should  be  included  in  the  greetings  script. 

Candidates  were  given  the  opportunity  to  take  bathroom  breaks.  The  movement  of  people 
in  the  testing  area  can  be  distracting  to  others.  A  constant  environment  must  exist  in  each  testing 
session  to  establish  sensible  metrics. 

There  were  occasions  when  the  Windows  environment  locked  and  the  candidate  needed 
assistance  to  continue  the  test.  The  testing  progranf's  restorability  is  extremely  poor.  The 
candidate  must  remember  which  session  they  were  on  when  the  computer  failed.  This  can  be 
problematic  because  test  session's  one  (1)  through  four  (4)  are  not  presented  in  the  same  order 
for  each  occurrence  of  the  test.  The  testing  program  should  remember  if  the  test  is  completed  or 
else  start  from  where  the  candidate  left  off. 

A  system  hardware  failure  was  also  experienced.  Having  the  software  installed  on  jump 
drives  allowed  the  switching  of  computers  to  be  more  efficient.  Replacing  the  hardware  and 
connecting  the  jump  drives  was  all  that  was  required  to  bring  the  testing  station  back  online. 
Downtime  was  no  more  the  10  minutes.  When  removing  a  computer  from  service,  make  sure  to 
label  it.  This  will  eliminate  the  risk  of  the  bad  machine  being  used  again  in  any  future  test 
sessions. 

A  highlighted  banner  at  the  end  of  the  exam  should  be  displayed  to  convey  the  test  is 
completed  and  wait  for  further  instructions.  The  test  indicated  to  raise  your  hand. 

Ethnic  background  and  race  needs  to  be  clearer  and  broader.  Perhaps  incorporate  an 
“Other”  category. 

Acronyms  need  to  be  spelled  out  initially.  This  is  an  Information  Technology  (IT) 
aptitude  test  and  not  everyone  may  know  what  IT  means. 


Data  Collection: 


Collection  of  data  for  this  experimental  test  was  a  manual  process.  The  jump  drives 
containing  the  test  software  were  disseminated  to  each  table  containing  four  (4)  test  stations. 
Each  test  station  received  a  unique  variation  of  the  test  or  Test  Form  ID.  For  example,  Table  one 
(1)  received  Test  Form  ID  A01,  B01,  C01,  and  D01.  The  jump  drives  are  logged-out  on  the  On- 
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Site  Jump  Drive  Tracking  Form  (Attachment  II)  as  the  drives  are  inserted  into  the  testing 
stations  USB  port.  A  scannable  ,. Information-Communication  Technology  Test  Development " 
(Attachment  III)  document  is  left  at  each  testing  station  for  the  candidate  to  complete.  The  pre¬ 
slugged  “ID  CODE”  on  the  scanable  document  is  logged  with  the  jump  drive"s  Test  Form  ID. 
This  links  the  candidate  to  the  item  responses.  As  the  candidate  completes  the  test  the  item 
response  to  each  question  is  written  to  the  jump  drive. 

After  the  last  testing  session  is  completed  the  jumps  drives  are  collected  and  the  test 
results  loaded  to  a  central  depository.  Since  each  computer  is  a  stand-alone  testing  station 
gathering  of  data  can  only  be  done  by  physically  collecting  the  individual  jump  drives.  When  the 
jump  drives  have  been  collected  an  automated  process  (GrabData  lCT.Bat)  is  initiated  to  collect 
the  data  from  the  jump  drive.  The  operator  inserts  each  jump  drive  and  initiates  the  .Bat  file.  The 
jump  drive  is  then  logged-in  on  the  On-Site  Jump  Drive  Tracking  Form.  This  process  is 
repeated  for  each  jump  drive  used  in  the  testing  session. 

This  data  collection  step  is  sufficient  enough  to  complete  the  task.  However,  a  more 
robust  interface  program  can  be  written  to  eliminate  the  „AutoPlay"  and  DOS  window.  There  are 
many  windows  being  initiated  when  the  .Bat  file  is  consecutively  run  one  hundred  (100)  times. 
When  under  program  control  the  multiple  windows  would  be  eliminated  because  the  interface 
program  will  manage  when  the  USB  device  is  ready,  the  item  responses  are  collected,  and  the 
device  is  ejected.  As  a  final  step,  the  operator  would  be  asked  to  end  the  program  or  insert 
another  jump  drive. 

Multiple  documents  were  used  to  manage  the  testing  environment.  The  first  document 
was  the  Test  Station  Configuration  Diagram.  This  diagram  contains  the  planned  seating 
arrangements  for  each  table  within  the  testing  environment.  The  second  document  was  the  On- 
Site  Jump  Drive  Tracking  Form.  This  document  manages  the  flow  of  jump  drives  from  the 
testing  station  to  data  collection  by  recording  the  Jump  Drive  Number  and  Subject  ID.  It  would 
be  beneficial  to  combine  both  Test  Station  Configuration  Diagram  and  On-Site  Jump  Drive 
Tracking  Form  into  one  comprehensive  document.  These  artifacts  together  fonn  a  sampling 
matrix  that  the  administrators  could  use  to  plan  the  test  site  and  distribution  of  test  materials,  (i.e. 
jump  drives  and  scanable  documents)  This  sampling  matrix  could  be  automatically  generated 
with  built-in  business  rules  for  distribution  and  labeling  of  Test  Forms.  In  addition,  this  process 
could  also  generate  pre-slugged  Jnformation-Communication  Technology  Test  Development" 
documents.  These  automated  steps  would  eliminate  inconsistent  distribution  of  test  materials  and 
provide  a  reiterative  process  that  can  be  used  across  all  projects. 
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Attachment  I:  Test  Station  Configuration  Diagram. 


1.  Each  outlet  highlighted  in  yellow  has  2  plugs.  A  surge  protector  with  6  outlets  is  plugged 
into  each  one.  This  allows  capability  for  12  laptops  to  be  plugged  in.  We  will  only  plug 
in  4  per  table  so  this  will  leave  2  outlets  open  per  table  in  order  to  utilize  extension 
cords  to  support  the  inner  tables. 

2.  Materials  needed:  25  Surge  protectors,  7  15  -  20  ft  extension  cords  to  support  100 
testing  stations.  Ten  (10)  laptops  will  be  used  for  backup. 
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Attachment  II:  On-Site  Jump  Drive  Tracking  Form 


On-Site  Jump  Drive  Tracking  Form 


Jump  Drive  # 

C01 
C02 
C03 
C04 
C05 
C06 
C07 
C08 
C09 
CIO 
C11 
Cl  2 
Cl  3 

_ C1_4 _ 

_ C1_5 _ 

Cl  6 
Cl  7 
Cl  8 
Cl  9 
C20 
C21 
C22 

_ C23 _ 

_ C24 _ 

C25 

C26 

C27 

_ C28 _ 

_ C29 _ 

C30 

DPI 

D02 

D03 

D04 

_ D05 _ 

_ D06 _ 

D07 
D08 
D09 
DIO 
D1 1 
D12 
D13 
D1 4 

_ D15 _ 

D1  6 
D17 
D18 
D19 
D20 
D21 
D22 

_ D23 _ 

D24 

D25 

D26 

D27 


out  in 
out  in 
out  in 
out  in 
out  in 
out  in 
out  in 


out  in 
out  in 
out  in 
out  in 
out  in 
out  in 
out  in 


D28  out  in 


D29  out  in 


D30 _ |  out  in 


out  in 
out  in 
out  in 
out  in 
out  in 
out  in 
out  in 


out  in 
out  in 
out  in 
out  in 
out  in 
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The  ID  Code  column  is  pre-slugged  with  a  four  (4)  digit  number  to  link  the  candidate  to  their 
item  responses. 
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